{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NILMTK Rapid Experimentation API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This notebook demonstrates the use of NILMTK's ExperimentAPI - a  new  NILMTK  interface which  allows NILMTK users to focus on which experiments to run rather than on the code required to run such experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to note that handing over so much flexibility to the user does require the user to be somewhat familiar with the data set, but this part of the process is supported by NILMTK as data exploration is simple and well documented. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets us start with a very simple experiment to demonstrate the use of the API for multiple appliances in a minimal use case. This experiment shows how the user can select the appliances in the dataset on which disaggregation is to be performed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing the API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nilmtk.api import API\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we import the required algorithms on which we wish to run the experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nilmtk.disaggregate import CO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we enter the values for the different parameters in the dictionary. Since we need multiple appliances, we enter the names of all the required appliances in the _'appliances'_ parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment1 = {\n",
    "  'power': {'mains': ['apparent','active'],'appliance': ['apparent','active']},\n",
    "  'sample_rate': 60,\n",
    "  'appliances': ['fridge','air conditioner', 'microwave'],\n",
    "  'methods': {\"CO\":CO({})},\n",
    "  'train': {    \n",
    "    'datasets': {\n",
    "        'Dataport': {\n",
    "            'path': 'data/dataport.hdf5',\n",
    "            'buildings': {\n",
    "                10: {\n",
    "                    'start_time': '2015-04-04',\n",
    "                    'end_time': '2015-04-06'\n",
    "                    }\n",
    "                }                \n",
    "            }\n",
    "        }\n",
    "    },\n",
    "  'test': {\n",
    "    'datasets': {\n",
    "        'Dataport': {\n",
    "            'path': 'data/dataport.hdf5',\n",
    "            'buildings': {\n",
    "                10: {\n",
    "                    'start_time': '2015-04-25',\n",
    "                    'end_time': '2015-04-26'\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        'metrics':['rmse']\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example experimental setup, we have set the _sample rate_ at 60Hz and use Combinatorial Optimisation to \n",
    "disaggregate the required appliances from building 10 in the dataport dataset with the _RMSE_ metric to measure the accuracy. We also specify the dates for training and testing\n",
    "\n",
    "Next we provide this experiment dictionary as input to the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Joint Testing for all algorithms\n",
      "Loading data for  Dataport  dataset\n",
      "Dropping missing values\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............  rmse  ..............wave'ioner'\n",
      "                         CO\n",
      "fridge            92.925652\n",
      "air conditioner  487.541665\n",
      "microwave        746.574343\n"
     ]
    }
   ],
   "source": [
    "api_results_experiment_1 = API(experiment1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe the prediction vs. truth graphs in the above cell. The accuracy metrics can be accessed using the following commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataport_10_rmse\n",
      "                         CO\n",
      "fridge            92.925652\n",
      "air conditioner  487.541665\n",
      "microwave        746.574343\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "errors_keys = api_results_experiment_1.errors_keys\n",
    "errors = api_results_experiment_1.errors\n",
    "for i in range(len(errors)):\n",
    "    print (errors_keys[i])\n",
    "    print (errors[i])\n",
    "    print (\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was a trivial experiment that only scratches the surface of the true potential of this API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next experiment we will run an incrementally more complex version of the above experiment. Here we will use multiple models to disaggregate the appliance readings with the models having their own sets of parameters which can be set by the users within the experimental dictionary in order to fine tune experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also import the required algorithms for the next experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nilmtk.disaggregate import FHMM_EXACT, Mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment2 = {\n",
    "  'power': {'mains': ['apparent','active'],'appliance': ['apparent','active']},\n",
    "  'sample_rate': 60,\n",
    "  'appliances': ['fridge','air conditioner', 'microwave'],\n",
    "  'methods': {\"Mean\":Mean({}),\"FHMM_EXACT\":FHMM_EXACT({'num_of_states':2}), \"CombinatorialOptimisation\":CO({})},\n",
    "  'train': {    \n",
    "    'datasets': {\n",
    "        'Dataport': {\n",
    "            'path': 'data/dataport.hdf5',\n",
    "            'buildings': {\n",
    "                10: {\n",
    "                    'start_time': '2015-04-04',\n",
    "                    'end_time': '2015-04-06'\n",
    "                    }\n",
    "                }                \n",
    "            }\n",
    "        }\n",
    "    },\n",
    "  'test': {\n",
    "    'datasets': {\n",
    "        'Datport': {\n",
    "            'path': 'data/dataport.hdf5',\n",
    "            'buildings': {\n",
    "                10: {\n",
    "                    'start_time': '2015-04-25',\n",
    "                    'end_time': '2015-04-26'\n",
    "                    }\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        'metrics':['mae', 'rmse']\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Joint Testing for all algorithms\n",
      "Loading data for  Datport  dataset\n",
      "Dropping missing values\n",
      "Generating predictions for : Mean\n",
      "Generating predictions for : FHMM\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............  mae  ..............owave'ioner'\n",
      "                       Mean  FHMM_EXACT  CombinatorialOptimisation\n",
      "fridge            45.440582   77.579147                  79.816048\n",
      "air conditioner  397.251617  370.581329                 274.858612\n",
      "microwave         36.396835  436.638092                 461.106995\n",
      "............  rmse  ..............\n",
      "                       Mean  FHMM_EXACT  CombinatorialOptimisation\n",
      "fridge            60.877859   93.722053                  93.047745\n",
      "air conditioner  607.580550  575.861339                 487.461264\n",
      "microwave        166.414246  668.473027                 746.621181\n"
     ]
    }
   ],
   "source": [
    "api_results_experiment_2 = API(experiment2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[                       Mean  FHMM_EXACT  CombinatorialOptimisation\n",
       " fridge            45.440582   77.579147                  79.816048\n",
       " air conditioner  397.251617  370.581329                 274.858612\n",
       " microwave         36.396835  436.638092                 461.106995,\n",
       "                        Mean  FHMM_EXACT  CombinatorialOptimisation\n",
       " fridge            60.877859   93.722053                  93.047745\n",
       " air conditioner  607.580550  575.861339                 487.461264\n",
       " microwave        166.414246  668.473027                 746.621181]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "api_results_experiment_2.errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datport_10_mae\n",
      "                       Mean  FHMM_EXACT  CombinatorialOptimisation\n",
      "fridge            45.440582   77.579147                  79.816048\n",
      "air conditioner  397.251617  370.581329                 274.858612\n",
      "microwave         36.396835  436.638092                 461.106995\n",
      "\n",
      "\n",
      "\n",
      "Datport_10_rmse\n",
      "                       Mean  FHMM_EXACT  CombinatorialOptimisation\n",
      "fridge            60.877859   93.722053                  93.047745\n",
      "air conditioner  607.580550  575.861339                 487.461264\n",
      "microwave        166.414246  668.473027                 746.621181\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "errors_keys = api_results_experiment_2.errors_keys\n",
    "errors = api_results_experiment_2.errors\n",
    "for i in range(len(errors)):\n",
    "    print (errors_keys[i])\n",
    "    print (errors[i])\n",
    "    print (\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The API makes running experiments extremely quick and efficient, with the emphasis on creating finely tuned reproducible experiments where model and parameter performances can be easily evaluated at a glance.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next iteration of this experiment, we introduce more parameters _chunksize_, _DROP_ALL_NANS_ and _artificial_aggregate_ and add another disaggregation algorithm (_Hart85_). We also train and test data from multiple buildings of the same dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also import the Hart algorithm for the next experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nilmtk.disaggregate import Hart85"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment3 = {\n",
    "  'power': {'mains': ['apparent','active'],'appliance': ['apparent','active']},\n",
    "  'sample_rate': 60,\n",
    "  'appliances': ['fridge','air conditioner','electric furnace','washing machine'],\n",
    "  'artificial_aggregate': True,\n",
    "  'chunksize': 20000,\n",
    "  'DROP_ALL_NANS': False,\n",
    "  'methods': {\"Mean\":Mean({}),\"Hart85\":Hart85({}), \"FHMM_EXACT\":FHMM_EXACT({'num_of_states':2}), \"CO\":CO({})},\n",
    "  'train': {    \n",
    "    'datasets': {\n",
    "      'Dataport': {\n",
    "        'path': 'data/dataport.hdf5',\n",
    "        'buildings': {\n",
    "          54: {\n",
    "            'start_time': '2015-01-28',\n",
    "            'end_time': '2015-02-12'\n",
    "          },\n",
    "          56: {\n",
    "            'start_time': '2015-01-28',\n",
    "            'end_time': '2015-02-12'\n",
    "          },\n",
    "          57: {\n",
    "            'start_time': '2015-04-30',\n",
    "            'end_time': '2015-05-14'\n",
    "          },\n",
    "                }                \n",
    "            }\n",
    "        }\n",
    "    },\n",
    "  'test': {\n",
    "    'datasets': {\n",
    "      'Datport': {\n",
    "        'path': 'data/dataport.hdf5',\n",
    "        'buildings': {\n",
    "          94: {\n",
    "            'start_time': '2015-04-30',\n",
    "            'end_time': '2015-05-07'\n",
    "          },\n",
    "          103: {\n",
    "            'start_time': '2014-01-26',\n",
    "            'end_time': '2014-02-03'\n",
    "          },\n",
    "          113: {\n",
    "            'start_time': '2015-04-30',\n",
    "            'end_time': '2015-05-07'\n",
    "          },\n",
    "        }\n",
    "      }\n",
    "    },\n",
    "        'metrics':['mae', 'rmse']\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Joint Testing for all algorithms\n",
      "Loading data for  Datport  dataset\n",
      "Dropping missing values\n",
      "Creating an Artificial Aggregate\n",
      "Generating predictions for : Mean\n",
      "Generating predictions for : Hart85\n",
      "Finding Edges, please wait ...\n",
      "Edge detection complete.\n",
      "Creating transition frame ...\n",
      "Transition frame created.\n",
      "Creating states frame ...\n",
      "States frame created.\n",
      "Finished.\n",
      "Generating predictions for : FHMM\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............ wer demand for 'washing machine'' mae  ..............\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             60.475124  26.411211   16.998512  58.242363\n",
      "air conditioner   133.285614   5.055853   22.030258  72.022423\n",
      "electric furnace   63.302246  88.532242   25.158630  10.183631\n",
      "washing machine     2.734084  82.629463    1.800099  12.294147\n",
      "............  rmse  ..............\n",
      "                        Mean      Hart85  FHMM_EXACT          CO\n",
      "fridge             75.940336   53.922593   63.108259   87.623677\n",
      "air conditioner   204.703034   40.421356  123.048184  187.298225\n",
      "electric furnace   82.090731  117.664927   93.185257   21.753245\n",
      "washing machine    10.896916  107.468327   11.097525   51.306552\n",
      "Dropping missing values\n",
      "Creating an Artificial Aggregate\n",
      "Generating predictions for : Mean\n",
      "Generating predictions for : Hart85\n",
      "Finding Edges, please wait ...\n",
      "Edge detection complete.\n",
      "Creating transition frame ...\n",
      "Transition frame created.\n",
      "Creating states frame ...\n",
      "States frame created.\n",
      "Finished.\n",
      "Generating predictions for : FHMM\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............  mae  ..............ing machine''\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             64.257904   3.944097    8.862674  45.464237\n",
      "air conditioner   132.515869   5.799566   29.786198  35.921093\n",
      "electric furnace   95.431465  74.070572   27.676476  45.702343\n",
      "washing machine     2.847503  29.716927    1.721615  21.721788\n",
      "............  rmse  ..............\n",
      "                        Mean      Hart85  FHMM_EXACT          CO\n",
      "fridge             65.094279   17.635512   25.227778  111.080550\n",
      "air conditioner   188.248236   66.696289  156.686939  157.176671\n",
      "electric furnace  132.698694  148.709435   66.595939  125.972037\n",
      "washing machine    11.848593   64.275367   11.525345   80.751974\n",
      "Dropping missing values\n",
      "Creating an Artificial Aggregate\n",
      "Generating predictions for : Mean\n",
      "Generating predictions for : Hart85\n",
      "Finding Edges, please wait ...\n",
      "Edge detection complete.\n",
      "Creating transition frame ...\n",
      "Transition frame created.\n",
      "Creating states frame ...\n",
      "States frame created.\n",
      "Finished.\n",
      "Generating predictions for : FHMM\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............  mae  ..............ing machine''\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             61.245720   6.915873    3.849008  32.178371\n",
      "air conditioner   111.373360   0.041766    4.968750  29.909920\n",
      "electric furnace   61.635025  41.444443   13.000000   1.269841\n",
      "washing machine     1.972475  41.444443    0.225000   0.491667\n",
      "............  rmse  ..............\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             61.491030  16.436289    7.007734  61.740105\n",
      "air conditioner   111.374399   0.484379    4.981663  59.116908\n",
      "electric furnace   61.635040  76.172319   13.000000   5.039526\n",
      "washing machine     1.972474  76.172319    2.846050   6.426508\n"
     ]
    }
   ],
   "source": [
    "api_results_experiment_3 = API(experiment3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datport_94_mae\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             60.475124  26.411211   16.998512  58.242363\n",
      "air conditioner   133.285614   5.055853   22.030258  72.022423\n",
      "electric furnace   63.302246  88.532242   25.158630  10.183631\n",
      "washing machine     2.734084  82.629463    1.800099  12.294147\n",
      "\n",
      "\n",
      "\n",
      "Datport_94_rmse\n",
      "                        Mean      Hart85  FHMM_EXACT          CO\n",
      "fridge             75.940336   53.922593   63.108259   87.623677\n",
      "air conditioner   204.703034   40.421356  123.048184  187.298225\n",
      "electric furnace   82.090731  117.664927   93.185257   21.753245\n",
      "washing machine    10.896916  107.468327   11.097525   51.306552\n",
      "\n",
      "\n",
      "\n",
      "Datport_103_mae\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             64.257904   3.944097    8.862674  45.464237\n",
      "air conditioner   132.515869   5.799566   29.786198  35.921093\n",
      "electric furnace   95.431465  74.070572   27.676476  45.702343\n",
      "washing machine     2.847503  29.716927    1.721615  21.721788\n",
      "\n",
      "\n",
      "\n",
      "Datport_103_rmse\n",
      "                        Mean      Hart85  FHMM_EXACT          CO\n",
      "fridge             65.094279   17.635512   25.227778  111.080550\n",
      "air conditioner   188.248236   66.696289  156.686939  157.176671\n",
      "electric furnace  132.698694  148.709435   66.595939  125.972037\n",
      "washing machine    11.848593   64.275367   11.525345   80.751974\n",
      "\n",
      "\n",
      "\n",
      "Datport_113_mae\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             61.245720   6.915873    3.849008  32.178371\n",
      "air conditioner   111.373360   0.041766    4.968750  29.909920\n",
      "electric furnace   61.635025  41.444443   13.000000   1.269841\n",
      "washing machine     1.972475  41.444443    0.225000   0.491667\n",
      "\n",
      "\n",
      "\n",
      "Datport_113_rmse\n",
      "                        Mean     Hart85  FHMM_EXACT         CO\n",
      "fridge             61.491030  16.436289    7.007734  61.740105\n",
      "air conditioner   111.374399   0.484379    4.981663  59.116908\n",
      "electric furnace   61.635040  76.172319   13.000000   5.039526\n",
      "washing machine     1.972474  76.172319    2.846050   6.426508\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "errors_keys = api_results_experiment_3.errors_keys\n",
    "errors = api_results_experiment_3.errors\n",
    "for i in range(len(errors)):\n",
    "    print (errors_keys[i])\n",
    "    print (errors[i])\n",
    "    print (\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of the above experiment are presented for every chunk per building in the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following experiment, we demonstrate how to run experiments across datasets, which was previously not possible. The important thing to pay attention to is that such datasets can only be trained and tested together as long as they have common appliances in homes with common _ac_types_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment4 = {\n",
    "  'power': {'mains': ['apparent','active'],'appliance': ['apparent','active']},\n",
    "  'sample_rate': 60,\n",
    "  'appliances': ['washing machine','fridge'],\n",
    "  'artificial_aggregate': True,\n",
    "  'chunksize': 20000,\n",
    "  'DROP_ALL_NANS': False,\n",
    "  'methods': {\"Mean\":Mean({}),\"Hart85\":Hart85({}), \"FHMM_EXACT\":FHMM_EXACT({'num_of_states':2}), 'CO':CO({})},   \n",
    "  'train': {\n",
    "    'datasets': {\n",
    "      'UKDALE': {\n",
    "        'path': 'C:/Users/Hp/Desktop/nilmtk-contrib/ukdale.h5',\n",
    "        'buildings': {\n",
    "              1: {\n",
    "                'start_time': '2017-01-05',\n",
    "                'end_time': '2017-03-05'\n",
    "              },          \n",
    "            }\n",
    "          },        \n",
    "        }\n",
    "      },    \n",
    "  'test': {\n",
    "    'datasets': {\n",
    "      'DRED': {\n",
    "        'path': 'C:/Users/Hp/Desktop/nilmtk-contrib/dred.h5',\n",
    "        'buildings': {\n",
    "              1: {\n",
    "                    'start_time': '2015-09-21',\n",
    "                    'end_time': '2015-10-01'\n",
    "          }\n",
    "        }\n",
    "      },\n",
    "      'REDD': {\n",
    "        'path': 'C:/Users/Hp/Desktop/nilmtk-contrib/redd.h5',\n",
    "        'buildings': {\n",
    "              1: {\n",
    "                    'start_time': '2011-04-17',\n",
    "                    'end_time': '2011-04-27'\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    },\n",
    "        'metrics':['mae', 'rmse']\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Joint Testing for all algorithms\n",
      "Loading data for  DRED  dataset\n",
      "Dropping missing values\n",
      "Creating an Artificial Aggregate\n",
      "Generating predictions for : Mean\n",
      "Generating predictions for : Hart85\n",
      "Finding Edges, please wait ...\n",
      "Edge detection complete.\n",
      "Creating transition frame ...\n",
      "Transition frame created.\n",
      "Creating states frame ...\n",
      "States frame created.\n",
      "Finished.\n",
      "Generating predictions for : FHMM\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............  mae  ..............ge' machine'\n",
      "                      Mean    Hart85  FHMM_EXACT         CO\n",
      "washing machine  25.636555  2.186584    2.276629  24.139162\n",
      "fridge           43.427475  7.185967    4.252556  25.250912\n",
      "............  rmse  ..............\n",
      "                      Mean     Hart85  FHMM_EXACT         CO\n",
      "washing machine  54.964733  29.918196   31.821257  50.603907\n",
      "fridge           45.180869  20.587223   10.595766  49.527368\n",
      "Loading data for  REDD  dataset\n",
      "Loading data for meter ElecMeterID(instance=2, building=1, dataset='REDD')     \n",
      "Done loading data all meters for this chunk.\n",
      "Loading data for meter ElecMeterID(instance=20, building=1, dataset='REDD')     \n",
      "Done loading data all meters for this chunk.\n",
      "Dropping missing values\n",
      "Creating an Artificial Aggregate\n",
      "Generating predictions for : Mean\n",
      "Generating predictions for : Hart85\n",
      "Finding Edges, please wait ...\n",
      "Edge detection complete.\n",
      "Creating transition frame ...\n",
      "Transition frame created.\n",
      "Creating states frame ...\n",
      "States frame created.\n",
      "Finished.\n",
      "Generating predictions for : FHMM\n",
      "Generating predictions for : CO\n",
      "...............CO disaggregate_chunk running.............\n",
      "............  mae  ..............ge' machine'\n",
      "                      Mean     Hart85  FHMM_EXACT         CO\n",
      "washing machine  47.244759   9.858242   20.557049  28.607571\n",
      "fridge           62.311844  53.618702   32.754597  32.168388\n",
      "............  rmse  ..............\n",
      "                       Mean      Hart85  FHMM_EXACT         CO\n",
      "washing machine  236.715613  114.865967  186.791693  78.342796\n",
      "fridge            86.095885  100.099604   61.811505  56.396517\n"
     ]
    }
   ],
   "source": [
    "api_results_experiment_4 = API(experiment4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DRED_1_mae\n",
      "                      Mean    Hart85  FHMM_EXACT         CO\n",
      "washing machine  25.636555  2.186584    2.276629  24.139162\n",
      "fridge           43.427475  7.185967    4.252556  25.250912\n",
      "\n",
      "\n",
      "\n",
      "DRED_1_rmse\n",
      "                      Mean     Hart85  FHMM_EXACT         CO\n",
      "washing machine  54.964733  29.918196   31.821257  50.603907\n",
      "fridge           45.180869  20.587223   10.595766  49.527368\n",
      "\n",
      "\n",
      "\n",
      "REDD_1_mae\n",
      "                      Mean     Hart85  FHMM_EXACT         CO\n",
      "washing machine  47.244759   9.858242   20.557049  28.607571\n",
      "fridge           62.311844  53.618702   32.754597  32.168388\n",
      "\n",
      "\n",
      "\n",
      "REDD_1_rmse\n",
      "                       Mean      Hart85  FHMM_EXACT         CO\n",
      "washing machine  236.715613  114.865967  186.791693  78.342796\n",
      "fridge            86.095885  100.099604   61.811505  56.396517\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "errors_keys = api_results_experiment_4.errors_keys\n",
    "errors = api_results_experiment_4.errors\n",
    "for i in range(len(errors)):\n",
    "    print (errors_keys[i])\n",
    "    print (errors[i])\n",
    "    print (\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like the above experiments, any user can set up other experiments very quickly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
